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^ Abstract' 



« The purpose of this paper is to suggest an approach^ that 

f ~ . 

can be used for integrating f ormati ve evaluatiaii - into externally 
mandated program reviews. The first step in this integration j^s 
to classify the mandat ed«»sprogram ^review criteria using the two 
dimensional model developed in this paper. This model is based 

on a crossr-classif icat ion of the ttfo major dimensions of program 

* it 
sequence and program participants • This, model can be used to 

> . A - * . 

examine externally mandated program review criteria /in order to 

determine aspects of the program,, that are riot jturrently being 

assessed. The next step is to develop cjrit eria^ that can used to 

enhance the utility br^the mandated evaluation ; for formative 

purposes . The model can be used to suggest crittexia for this 

formative purpose. The model is illustrated using data from 

medi'um sized urban university. 



. ?HE ROLE OF FORMATIVE EVALUATION IN EXTERNALLY MANDATED 

■ • ■ a ■ 

PROGRAM REVIEWS IN HIGHER EDUCATION' ,' 

o The purpose of this paper is to suggest a ; model that can be 
used for integrating the formative evaluation of academic 
programs with .the requirements o£ man^J^ed program reviews in 
higher education. Externally mandated reviews ard program 
Evaluations that- are undertaken at the request of ■an outside 

agency or organization. Statewide program reviews that are 

^ > . . ■ ;■ .. . .* •-• 

typicalTy required by governing boards in higher education and 
also accreditation activities would all fall into the "category 
of externally /mandate^ program review. ~ • , 

Externally mandated reviews are generally concerned with 
the summative evaluation of academic programs at the university. 
Sunfmative ^valuations focusr primarily on the filial/ outcomes of * 
the academic programs. A> goo d*' example of the types of outcomes 
that, can /be used for su^native evaluation in higher education is 
the ?et /proposed by the National Center for Higher Education ' 

Management Systems (Lenning , 1977) • According to Lennings ( 1977) , 

. ■ * * j 

t ; 

outcome^ are the end results of the processes that occur within 
post secondary education institutions and programs. This concern 
with "end results" can be contrasted with formative' evaluations 
of academic programs in higher ^education . 

Formative evaluations of academic programs shotuld assess 
criteria that can be used to provide feedback to the program 
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participants in order to improve f ^he program. "This idea of *• 
feedback and correctives is an essential ingredient of*' formative 
evaluations • The ^distinction between formative and summative V >' 
evaluation was initially introduced by Bloom,. Hastings aud 

Madaus (1971) in relation to the evaluation of student learning/ 

c> ' ■ - ■ . * • •* 

We feel that this idea can be meaningfully applied to program 
reviews in higher education as wqII. Our use of the term is moire 
general with specific emphases on the feedback and corrective 
aspects of the evaluation . . 

o , The ob*jecti-ves of formative and ; summative evaluations are 
different, but no t ^necessarily contradictory. In this paper we 
have' combined the goals' of each type of evaluation. The need for 
this integration is based on the notion that externally mandated 
reviews clan be made more productive, .if the goals of formative 
evaluation can also be met concurrently during the same 
■evaluation. The method used to develop criteria for both 
formative and summative evaluation is based on the model 
described in this paper. 

The first step in increasing the formative, evaluation 
aspec ts of an "external ly : ''mandated''" program; review is to assess 
the criteria that are currently being used as a ,part of the 
summative evaluation. This ^can be* accomplished by categorizing 
the mandated criteria into a* simple two ^dimensional model. This . 
model is created by the croSs-classif ication of the major stages 
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t4iat any academic, program must pass through (input, pjrocess, and 
output) with this major par ti cpant s in the decisions" concerning 
tne academic program critejfi a ( student s , faculty, and 
administrators) • This model for classifying progranr review 



criteria will be discussed in depth in the next section • ' 

'* Once the mandated criteria ha ve .been 'classified , the next 
step is to develop criteria that can be used to enhance the 
utility of- ttfe program* evaluation for formative purposes. The 
mo3t important criteria: for ^ the formative evaluation of an 
academic program would be in the process cells and then perhaps 
the input cells, Tjtjjejsie j cells in the model for assessing 
potential pro grant review cri teria. would be most crucial because 
of their potential for alterdbility . Of course specific .criteria 
would vary in 'terms: of alter ability The/ identification and 
quality of the process ^criteria are crucial from a formative 
evaluation ;^p£rspec ti ve. The process criteria define .what., is 
actually happening -in the program and also the technology that 
is being usej Jto implement the planned changes of the inptft 
variables in^p- outcomes. Alt hough the input variables may be 
important from a formative perspective, these 1 criteria will 

generally no,t have as much potential for al terability . 

v H ■ ' * 

In summary^ the purpase of . this paper is to propose an 

'}..•,'" 

approach that can be used to combine formative evaluation with 

. i ; ' 

f ' • , I"..' • ■ ■ -. ■ . • 

the summative evaluations that are mandated by external, 



ft- 
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agencies.. In order to accomplish this * inter gration , a -model i,s. 



proposed for examining the mandated criteria in order to develop 

' * * ■' * . ■ ' * ' 

new criteria that can be used in conjunction with the mandated 

program evaluation', The" model for examining program revie^ 

criteria can b e • used to identify weaknesses in the ex is ting 

, \ V ■ - * : • ' ; ' , • - . ' . . 

structure of the summative outcomes that limittheir usefulness 
for formative evaluation. It is anticipated that the" criteria 
used in mandated reviews will tend to be in the outcome 
categor-y. Th<»^ rationale and description of the model for 
examining program re^HfBV- criteria is described in phe next 




section* Tfrife mj^fel will then 'beitfr^plied to' examiae a set of « 
criteria tfha-t were used in an actual program review at an urban 
un^versit : y. j . •? « * 

". . . '*i ./ ^ Description of the model 

/ The model -for examining mandated program reyiew criteria 
proposed in this paper is v composed of two major dimensions. The 

" — : ;. .. / . ..; . .. . ^ ... | 

first djjjiension represents the maj'br stages that dny program ' 
activity must pass through*- input process and output. The 

/. : ' - v •. \ • - 

second dimension is identified^ by the par ticipan tsu/in the 
decisions Concerning academic pr ograms - that is , students. , 
faculty and administrators. Wg recognize the presence of o'ther 
influences on the - institution and program,' such as governing' 
boards , \ legi si a tu res , communities , alumni , employers, and med iq , 
but our model, holds that these influences' are* mediated through- 
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■ ■* .*7 , , ■' 

) ■» • " • • ; ■ . 

• •. ■ - ■ • • ■ ; . /■. • ' \, 

one Qr more of the three m^or Institutional participant groups 
identified • above. The two^ dimensions can be CTOss-classifi'ed to 
yield a 3 x 3 'jn^tri* with nine cells. .1 ' 

In order to assign program review criteria to one of 'these 
ctells two prin/i^les can.be* used.. Th£ principle for assigning, * 
criteria to the first dimension of program sequence ' depends on v ^ 



he 



whether or n^t the, criteria is present at the onset o£~>^ 
program (input), used in the implementation of tiie program 

; * V > : - ^ ■ ■ * > ■ " ■ 

(process ), dr represents the outcome of th*e program (output) . 

The principle for assigning criteria to the second dimension is 

based on a consideration of v which of these thfree major groups 

has primary control*^over the act ion^j:epr4sented by the criteria. 

For example, StTj/tlents en ter , the program with certain* 

characteristics (e.g., ability, motivation, previous 

J0 ^ m ' ' ' 0 

achievement).- Thes^chafact eristics Vould b^ classified as 

student/input characteristics-. Another example is the 

arrangements that ar e , pro vided \ by the program -for accommodating 

diyerse student needs (e.g., -day ^are facilities, off-campus 

courses, and weekend courses). - - These characteristics are 

primarily under, the control of administrators, and relate to ^ 

^program prccess; jthese would be' ^placed in the * 

administration/ process cell. - 

v Each of tlie cells produced by the cross-classification of 

the two d;Lmen$it>ns - program sequence and pr agr'am part icipants - 
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can define a set of criteria that can be used for program review 
by an outside agency and/or developed by a university in order 
to, supplement the information required by \ eaxh^j^roup. These 
criteria cap be used by interested external or university based 
groups to examine program quality, efficiency, or other goals. 
The t choice of criteria and the cell which it represents is ve^y 



important because the planning and decisi'on making activities « 
that Result from the program review will vary among the^various 
parties depending on the cell^ which are used and the evaluation 
. criteria \generated within cells. Although the cells can be 
separated 'analytically , the criteria and the cells within the 
model are in reality dynamically inter-connected. 

Y * ■ 

The classification of program review criteria . into this 
model can highlight the cell or cells which are currently 
receiving the greatest emphasis in the external review. It. can 
also identify empty** cells, duplication, and criteria not j 
currently being used in the external program review activities 
which the University may wapt to include in its own program 
review. ^ 

Since each cell can be examined independently, a cell can 
generate its own definition of program quality. Ideally, for 
formative evaluations all of the cells should be represented in 
the program review and each cell' should contain multiple 
criteria. The assessment of program quality should not be based 
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.on any single criterion or any single cell, z The definition of 
quality should be multi-dimensional in the sense, that there 
should be multiple cri teria f within cells and also 
multi-dimensional in the sense that several cells are used. 

It. is also important to note that the cells are not 
necessarily of equal importance to either group. In fact 
different cells in the model can be assigned different levels of 
importance depending on the decision-making and planningc^on text 
of *the program. By. examining through this model the criteria 
proposed by an outside agency, it is possible to draw inferences 
-concerning the value assigned to each cell by the board. Such 
inferences can affect internal planning and policy, 

"Making -the relative importance and^ value of the various 
cells af external program review explicit is very useful to 
administrators and e valuators wi thin the university' who are 
typically confronted with a wide variety of seemingly unrelated 
program review criteria. Making the structure underlying the 
criteria- explicit enables them to place more weight on, the 
collection, and analyses of data for cri teria that are t>f maior 

,. * . . . .... '. /.. 

importance to the outside ageniy and the university. It can 
also be used to suggest?' supplemental vprogram information that' 

I 

the university may want k to -obtain 'jbn conjunction with the 
criteria * mandated by the state boat* d * s program review 
activities, . '„ i 
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". - .... / ' ■ ■ ' ..' 

s • 

In spite of the^ increasing "use of state mandated program 
reviews, relatively little time and effort has been . invested in 
examining the structure of currently used criteria fpr. their 



A. 



concurrent forma ti ve use within institutions. The selection of 
criteria to a large extent determines the priorities and policy 
that can_.be derived from the evaluation and used in planning; 
both /statewide planning and within university ^planning (Bloom, 
et al\ , 1971,; Dillon and Starkman, 1981; Dressel, 1976 J JStarkman 
aivd Bellis, 1977; Tyler, 1949). 

In ordeP>\to illustrate the classification of mandated 
criteria into tjhe ppbposed model, a set of criteria pro^ftrsed a 
board of higher/education will be examined. These criteria were 
used in the program evaluations of an urtfan university. 

Case slTudy 

* - * * 

: « The example used in the case study was conducted at a 

* t 

medium sized uni versity (7 f 000 undergraduate and graduate 
students). The student body is primarily blacky (75%), and^ihere 
are almost twice as many female students as Kale students. It! 
is a commuter university and most of the students come from the 
metropolitan area • 

The- university is operated' under the governance of a boafd 
of governors of state colleges and universities. The bo&rd of 
governprs has a responsibility to "...review periodically all 
existing programs of instruction, research and public service at 
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the ? state universities and to advise .the appropriate board of 
control if v the contribution of each program is not educationally 
and economically justified" . The program reviews which were 
developed to meet this responsibility are designed primarily to 
address^tate-level concerns. The gp.verning board alao 
recognizes that 11 ... program re views* support internal, university 
decisions related to planning, resource allpcatton , program 
impro vement t and accreditation requirements" . 

While reviewing the criteria-delected, by the' governing 
board for program re view, f it was recognized that the criteria 
were not assessing several program components which woulil be of 
significance to the .Institution for Normative evaluation, 
although 0 not necessarily of interest to the state board. In 
fact mfcst of the criteria seemed to fall into a Small subset of 
areas. In order to systematically examine this observation, the 
model discussed in the previftus section was developed; Each of 1 
the* criteria for-program review were exam^d and classified 
into the model for examining mandated criteria. ' The results are 
-shown** in Table L * - 

Overall tnere were 32 criteria used in the program review. 
Approximately 28 percent of Jthe criteria were classified into 
student/output cell. This was ti\e most heavily weighted cell. 
Educational aspirations, Satisfaction with the pr^granl and 
student gra de final point averages are -examples of the typical 

... • Ni 
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\" ' ; ••}*•'•"•• .C.:- ■ V'V : - • • 

. v- .. ■ ' • : : • ••••• " '. . ' \-. ' • , 

criteria included in this cell , d v v 

The. administrator /output cell was thp next most heavily 
weighted cell^ with' 18. 8 percent of the. criteria being classified 
into this category .Some of the typical criteria that occurred 4 
in this category were number of co ses taught., unit costs, ^arid 
accreditation of the program . j ■ •. v' 

The third largest cell was the admini$£.rat or/process cell 

with approximately 16 petcent of the criteria iassessijig 

•'. • ."»■-'." • ■'. 

behaviors from this category • Arrangements for tf of f -campus 
courses, academic support and library support are aril examples 
• of .this cell, \S- : * m . : " 7 :.]■) ■ :/V ''c^;, 

... The faculty/input -cell included 12.5 percent of the 

criteria (e.g., program objectives, number of part-time faculty 
and percent tenured) . Ther^i%re:=9. A percent of the criteria in 
the student /input cell (e . g. , student backgrdUncT 
characteristics; high school .ranks and entry exam results). 
Finally, there were 6.2 percent o^f- the criteria in the^ 
administrator /input cell (e.g., enrollment data and resource 
;. requirements) • j ) 

One of the most ^ntelresting findings is the 'facA* that the 
student/ process and the f acuity/process cells were not ■ assessejl^ 
by* -any criteria. In general when educators think about program 

evaluation, these are' the types of actiYiteSi,-that v are typically 

• v J . v : ^ T -"" - ■"■"'■-V'. ' < v • ■ 

usei to dfefine the quality of a program, at least from a 1 



Formative Evaluation in Higher Education Page 11 



formative evaluation point of view, 

The mo.ffel clearly highlight's the emphasis on input "and 
output activities with 56 percent being outcome criteria, ^8 
percent input criteria and 16 percent process criteria. The 
criteria are about equally distributed between student and 
administrator categories with 38 percent and 41, percent 
^respectively ; approximately . 22 percent of the criteria fall 
under the faculty category • While these emphases* may be most 
appropriate for externally based decisions about programs, the 

t&der-repr esented categories in the example are equaLly 

•' • ■ ■ ■ • . • ■ 

. . ■ ■ '•' / . . 

significant to internal planners, / 

Based on this assessment of the criteria in ierms of the 

model described in this pap'er, several activities are currently 

7 ' • . * / ■ ' 

underway to*develop criteria that can be used/ to assess the 
mpty cells student/process and faculty/process. Some of the 




criteria that ate being considered are participation in 
classroom discussions, attendance, a^id the use of learning 
supports and counseling services • All /of 6 these criteria which 
would be in the st udent/proces's^ cell/. 

• • • / ' ' v... * : s/fi;: \ . 

. Discussion , 

The purpose of this paper ^as to present a model that can 

be used to integrate formative /evaluation criteria into ongoing 

* - i : ' ' 

mandated summativje evaluations. An important part of this 
Approach involves the development of a model that can be used to 



.V) ... f /'■ ■ -■ ■ r >' 
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highlight the criteria currently used in any, evaluation of * 
academic programs in higher education ; T # hroug*h the detailed * 
assessment of program review criteria and the interrelations 
,b'-etwe£n these criteria, a view* of state-wide and institutional 
priorities can be developed . The model proposed in this paper A 
can be used to stimulate an examination "of the possible 
relationships between criteria. Some possible questions dre as 
follows:* Are there correlations between faculty input 
characteristics and student process? Is there *a relationship 

. y , . . 

between faculty input and faculty process? What is the impact of 
faculty input and process on student outcomes ? What would 
happen to student outcomes, if we changed faculty input"? Such 
questions could lead to a greater understanding of the process 
f of schooling in higher education • ~" r "' r '[ 

One of the consequences of using this model in d\ir "case N* 
study" was the identification of the great emphasis placed by 
the governing board on student /output criteria and 
administrator /output criteria, while two other possible program 
review criteria areas - student/process and ' faculty/process ~ 
were not assessed at ail. \ 

Since outside agencies are primarily concerned with 
summative Evaluation , it is not too surprising that in our case 
study the criteria are not equally represented in each cell in 
the model. It was apparent that the governing board was not 
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*• * ■ , 

primarily concerned with process critieria that might have ^ 
explained outcome results obtained from th^program undergoing^ 
review. However, from a feedback/corrective paint of view,-/ the 
criteria .areas that could provide the most useful information 
for revising a program, .if 'that seemed appropriate , would tend 
to fall in the process cells - student, faculty, and 
administrator . Two of the process areas ( student and faculty) 
that would be very useful for the formative evaluation of an 
academic program were not even evaluated within the current 
program review structure. The process cells would represent 
criteria that should be of concern in most formative' program 
evaluations; these are the variables that define what the V 
faculty, students and administrators are actually doing and is 
typically considered the "academic program" . . 

In developing criteria that are appropriate lor all the 
programs and universities within a system, the governing board 
has identified areas that would be the most general and thereby 
applicable to a wide variety: of different programs. 
Unfortunately , these reviews lose some of their usefulness 
within each university because of this generality. Since the 
externally- mandated program reviews represent a major evaluation 
activity by the universities within the state, the 
administrators within the university that are responsible for 
the program evaluations should- beVaware of the impact of the 
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criteria that are chosen. / " : >, : . 

' ' '. ' ' ' - U| ; ' • * if. 

The response madefy the^uriivef sit y used in the case study 
was to develop criteria that could be <uaed to assess . areas' not 
currently tapped by the board criteria in order to maximize the 
internal utility of these revieVs f or^f brmati ve evaluation. The 
view adopted was that since the programs Were currently 
undergoing an externally -mandated program review, the 
supplemental information' designed to -address specif ically the' \ 
wi thin-university planning, management 'and decision-making nee^ds 
should become an integral part of the process and not merely 
ancillary. 
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Table 1. Classification of Evaluation Criteria 



Sequence Participants 



Sadple criteria 



Percent 



Input Students '-. 

" > Faculty 

Administrators * 



Process 



Output 



* Students 
Faculty 
Administrators 
V 



Studeifts 
Faculty 
Administrators 



Student c|iaf£ct eristics 
Program objectives \ 
Program resources 



:uxty 



Time on-fcasl^ 

Student /faculty interaction 
Support services 



Satisfaction- with program 
6rant dollar's . 
Unit/instructional costs 



9.4' 
12.5 
6.2 

28.1 

0.0 
0.0 
15.6 

15.6 

28.1 
9.4 
18.8 

56. 3 



Note - Percentages represent the proportion of criteria in each 
cell out of total of 32 criteria. 




